An Evaluation Framework Based on Gold Standard Models for ..

This paper presents a weak supervised evaluation framework for definition question answering (DefQA) called Solon. It automatically evaluates a set of DefQA systems using existing human definitions as gold standard models. This allows the framework to overcome known limitations of the evaluation methods in the state of the art with the advantage that it is less supervised. In addition, Solon adapts its configuration for each specific DefQA task, thus rendering a good evaluation procedure. The results obtained in our experiments show that Solon is able to detect the best systems and to score them accordingly, with state of the art performance.